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ABSTRACT ^ < " ... f 

f \ As more/psychojogists become involved in program 

evaluation, program evaluation courses have been developed by many 
psychology departments* Psychology .graduate students (N-19) 
.participated .in a\semester-long simulation ^cburse. Students divided 
into small groups/ and each group, submitted a written proposal for. 
evaluating ttfe counse linage r vice* of a fictional community agedcy* * 
Students^xould quest^n^xbe agency director, role-piayed ay - the - 
instructor, Vo obtain evaluation-relevant information which had been 
intentionally oAit^ed from^ descriptive materials^boiit" the agency. A 
second proposal was also written and submitted by 'each , groups The 
groups 'then analyzed a, fictitious set of pfe-and post-measures 
describing t v he. psychosocial adjustment ^of three client- subgroups- and * 
prepared a comprehensive final evaluation report that included the 
analysis results* Data relevant to the effectiveness of the 
simulation w&s generated from gues^ioniair^es completed -by 
participants, questions asked by the 'groups, the proposals and final 
report, and the instructors informal observations* Both t the., 
performance bf the 'students during the -si1ililati*on and their 
subsequent evaluation of th£ /'simulation effectiveness indicated that*, 
th^^mula^tion wate involving ^and educational.* The findings suggest 
thtfrteach'ing evaluation research via simulation nee4 not be confined 
to 'short-term courses* (Author/NBB) i 
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Teaching* Evaluation Research 
via a Semester-Long Simulation 



In recent years a growing number of psychologists especially 
those in' the field of community psychology, have become involved 
in program evaluation (e.g., Cowen & listen, ^1980} Dowell & Kriesel, 



1981; Guttentag &Struening, 1975; Lachenmeyer., 1980; Perloff & 
Perloff, .1977; Spielbfercjer, Piacentey & Hobf^l, 1976; Stenmark, . « 
1977). Not surprisingly, this trend has been accompanied by the 
development of program evaluation courses in psychology departments 
across the country. ideally i these courses would always provide 
students with the opportunity, to supplement their clasVoo^learning 
wi^h some, form of participation in an actual progranf^evaluation . 
What options are available to the instructor, hdwever, when the , 
opportunity to evaluate a "real world" program does not exist for* . 
all' of the class? !One possibility" is to f&ve. students engage in a'* 

"simulated program evaluation. Northman (Note 1) reports success 
with such r an'approac-h when it was used as. part of a one-week ' insti- 
tute in program, planning and evaluation* attended by human service 

-administrators, clinicians, and' graduate students. The present 
paper describes a more elaborate simulation employed in a semester- 

• » * ' ' ' 

long graduate course in evaluation research. 



' \ s x ' Methdd '* ' 



Participant? ' * * ' • * . '\ ' \ . 

Nineteen students in anM.A. program ih clinical ^community . 
psychology participated '. in the simu^a\ion, which was conducted as 



part of an evaluation research course. .< There were also eight 
students in the^ course who tfad the opportunity ; to evaluate real 
programs. These individuals did not participate. in the simulation 
and their -experiences will not 'be focused on in this paper. 
Procedure 

Phase i (weeks 1-5) ; Early in the. semester students^sorted 
themselves into groups of three or four members each for the- simula- 
tion. A one-page 'description of the "Midtown Multi-Service Center 
a fictitious. community agency, was given 'to students in each of the 
five groups that were formed. 1 Intentionally omitted from this 
description was information which a skilled evalua tor would probably 
wish to examine before, generating an evaluation j&ari (e.'g-, an organ 
izational chart depicting the Center's structure in detail)." Most 
of the description dealt with the Center's counseling service, which 

the organization "s Board- of Directors was asking. each group to 

« . * • •* 

evaluate. , , . . 

^he first task, for each group was to submit to the Beafcd a 
written proposal (8-10 pages) for evaluating the counseling service. 
Each group was instructed to address the following questions in its 

» * 

proposal: * , . 

, lt • At a general level, how should the- evaluation, task be' § 

conceptual^ zed? : — ■ * r 

2. What special evaluation problems ate associated, vd^h this 
task and/or, setting, and how might these problems be handled? . \ 
. -3 .- What' methods should be used to determine the goals of -the . 

counseling service?* *" . 

# 4 # Wtfat criteria should be used to v Select a .subset of these « 

\ . • 

goals upon which the evaluation might focus? , . v 



5. Vjfhat is an appropriate research design, for assessing the 
counseling service's performance with respect to a given subset of 

these -goals? " - - 

6. To what extent is it feasible- and^appropriate to evaluate 

f , ' 

the counseling- service on the dimensions of effort,, efficiency, 
process, and adequacy of ^performance (Suchman, 1967, pp'. 60-71)? 

In the course of preparing Its proposal each group could submit 
five questions in writing to the Center's Executive Director, who 
was role-played by the course's instructor. It was the instructor's 
expectation that questions- would focus on information which was 
missing f^om the one-page description of the Center but which class 
lectures Ijad indicated was relevant to preparing an evaluation __ 

proposal. i ' 

For, the purpose of responding to the questions and clarifying ' 
any misunderstandings concerning these responses, a meeting was- held 
between .each group and the director. Some answers could simply be 
given orally, while others were sufficiently complex and/or detailed 

1 i 

* • I 

that a written reply was required. . „ ^ — _ * 

Phase 2 (Weeks 6-9) • After the proposals were evaluated by • 
the instructor and returned, a second' proposal was requested. This 
proposal .was to contain the following: ^ ^ 

* 1. A more detailed description of tjhe experimental or qua.si- . 

experimental evaluation research design wh.ich the, group reconunehded 
«...«.«• ', '» 

in its -initial 'prdposal-. (If* the group was unhappy with the first. 

design, it c'ould modify 'it for the second proposal.), This .required 

a discussion* of : . . • • 

<• . - ' * .i 

• * ■ a. -hovr the design handled each of the major threats^ to* •'- 

validity outlined- by Campbell and Stanley (1966); ^ , . 



b. how blocking, concomitant variables, and/or factorial 

design would be employeciUtif at all); - 

c. how subjects would be assigned, to experimental and 

control conditions : and ' , . • 

* , ' ' . 

d. the time frame ' and sample size ,for the study. 

• • • • * ' A -1 

,2. A general discussion of the process and outcome measures 

to he. used in the evaluation. 

3.. 'A one-page questionnaire constructed by the group which 
would assess the. Board of Directors' perceptions of the counseling 
service on iss«es relevant to the. evaluation. 

v • 

4. A discussion of how the data emerging from the evaluation 

*• 

might be used in agency decision-making. - 

. As was the case with the simulation's first phase, each group ♦ 
could address five questions to the director. * 

< t Phafse 2 (Week's 10-14). . . The simulation's final stage presented 

.the- groups with two tasks. First, each group analyzed a fictitious 

set of £re- and post-measures prepared^by the instructor describing 

the psychosocial adjustment of three subgroups (depressed,, psychotic,- 

•anxious) of the* counseling/service's clients.. (All participants in 1 

the' simulation received the same data.) The data were constructed 

< - . • 

so that each subgroup exhibited a different .pattern of improvement/ 

• - * ■•• 

honimprovement. " * 

Second, each group prepared a comprehensive final evaluation . t 
' repc-rt for-the BqarJ^wh^ch included the results of the preceding 

analysis- as one c«nponent7^n~^ts^report each group was instructed 

•'*•'. it- * '. 

. to >a^e whatever recommendations/considered appropriate to enhance 

* • . ' ■ • » " • . K ' ■'- - - * 

* thV" functioning, Of .the counseling Service.. Once again „ five-questions-. 

^we^ealloWed. 'The simulation ended wi*h the instructor • s* evaluation^ 



of thesle reports. / . ^ 



Results ^ 
\ — . ■ * 



Data relevant to tfie~ef fectiveness of the simulation come from 
four major sources: an anonymous* questionnaire completed by* parti- 
cipant's after the course ended; the questions asked by the groups; 
the proposals and final, reports'; and the Instructor f s ^nformal 
observations* 

Questionnaire Responses / 1 

Responses to Likert-type\i terns on the questionnaire indicated 
that students evaluated the simulation quite -positively • ' Specifi- 
cally, the simulation was seen as .a, well-organized, interesting, 



and effective teaching devitfe^vrfiich > contributed significantly to 
the' quality of the course (See Appendix A, items 1-4^) • /The omission 
of relevant information from the Multi-Service Center ^description, 
the procedure* of allowing; gtoups^to question the director, and the 
requirement of three papers during the. simulation were all viewed 

as pedagogically useful (see Appendix A, 'items' 5-7). • . 

* ' * ■ • . • * -< .-•' • * • 

Participants believed that the simulation clearly succeeded in 

"giving them the opportunity to: * 1. appiy in an evaluation 'setting 

the approaches -and techniques discussed in clas^t, and 2. a^prec^a-te 

"f^rst-hand 1 ' some of tJ\e difficulties involvjjH{in doing an evalus- 

< — j> 

.tion (see Appendix' B, items 1, 2). With Respect to the goal of 

having students experience the processes involved in working. on arr • 

evaluation team, the simulatipn >fared less, we 1.1 : * half of t^he 

respofitients Jiidged the simulation to only be "somewhat" successful 

in achie ving that* objective (see Appendix B',~item 3)#-. * / 



'It should be noted that although" the participants tended to '.. 
^believe that the simulation resulted in- less- learning for them than 
" participating in a real" evaluation would have, they also thought 
the former technique allied for more fair and equitable* grading of 
'-students- than the latter (see Appendix A, items 8, 9). ^ 

When asked in an open-ended question, to indicate what was the 
most valuable aspect of the simulation for them, most participants 
. pointed- either to the opportunity to apply concepts learned in' 

class or, more specifically, to the challenge involved in developing 
'an appropriate research design for their proposal. When the question 
focused on the least valuable aspect "of the simulation, the most 
frequent response dealt with the statistical analysis that* students 

had to" perform on the data set provided them by the instructor . . . 

' ( • ' ' ... » 

Group Questions ' , 

' "' The groups generally made good use of their allotted questions . 
to focus on the- information gaps characterizing the description of 
the setting. Inquiries routinely dealt with such issues as the 
Center's organizational structure," its official. mandate, the demo- 
graphic characteristics of the counseling service' s. clientele, the 
. range of "client problems, encountered, and the procedures . f or assigning 

.clients to staff. . * - 

A • • * * 

. On occasion the information requested (e.g..,' the Center's annual . 

•/ . ft •, • ' '• ' 

report) exceeded in scope what the instructor could provide. There 

' were. also' times when, either deliberately or inadvertently, groups 

presented as one question what actually were two clearly separable 

■inquiries. In. . ttaese instances. the instructor would respond by drawing 

the group's attention to the problem and asking them either to with- 



draw one of the 'questions or to present them' as two distinct items.; 

• ' • -\ ' ' * . • . 

Proposals and Final Reports t 

■ - • . 
• The quality of .the proppsals and final reports Varied from group 

to group i 'ranging from acceptable but mediocre to outstanding. Not • 

surprisingly, there seemed Jto* be a positive relationship between the 

quality of the questions a group asked during a giverwphase . of the 

.simulation and the quality of the 'proposal or report it subsequently 

produced.. Tn addition, groups primarily composed of individuals who 

* * „ 

excelled in traditional classroom activities (e.g., tests) also tended 
•to do well in the simulation.. In assessing the significance of these 
relationships- caution needs to be exercised, however, -given that all 
evaluations of student performance, emanated-" from the, same source, • 
i.e.,' the 'instructor. , . . . . 

I nformal Observations , ' " • . ■ 

— ' — ; * ■ * • ' S" 

Students*, appeared t'<5, become - quite involved in the simulation 
and devoted significantly more out-of-class time to, it than the- 
instructor had expected they would. , While this latter occurrence 
may have been partially due to .the simulation' requiring more -work 
on ti}e part of ■ students "than had beep intended, the. students' 
perception of their task as a challenging" and^ intriguing one. also 
seemed to be a "major contributing factor. - % a 

A moderately competitive 'atmosphere^developed among the groups, 
■Sj • • ... • W " • 

accompanied by expressions- of. anxiety^ and frustration (e.g.^'Tob 

many things a^e bey orid our control here l»; M We*don« t -have enough ^ 

v infprmationl'V "There's never enough timel") familiar to anyone. 

who. has been- a fledgling evaluator in the rmal world. In addition, 



throughout^ the simulation the groups could be observed grappling . 
with 1 a variety of process issues (e.g.; methods of decision-making, 
l^itra-group conflict * communication) relevant to small-^group 
dynamics- - ' . % > "7 

■ 

- ^ Discussion . ; 

The findings of this study suggest that teaching* evaluation 
^research, via simulation need not be confined to short-term courses. 
It must be emphasized, however, that>$onducting a simulated program 

* . • * • * * • 

evaluation is no substitute for conducting a real one, and the first 
/priority of instructors should be to provide their students with the 
^latter experience. ^ p ■ 

Plr-h^ps" the idea rearrangement, would be one where a simulated 
evaluation was conducted during the first' semester of a two-semester 
•program eva.luatio>r course, with a real evaluation taking place -during 
the second. semester. Such an approach would result in students being 
better prepared for "the Veal evaluation thata if they weS exposed 
both to. the' course content and to the real evaluation simultaneously,, 
as„is oftert the' case. Indeed,* in a course involving a real program 
evaluation it would be instructive to have three groups of. students 
whose performances- could be compared : ; the first group would be one 
which had received no training in evaluation prior -to the semester 
in question; -the. second would have had a lecture, course only; and 
the third would have h*ad a lecture course which included a' simulation. 

• Regardless of how a simulation is integrated into a course \ 
sequence in program evaluation, tKe simulation described in this 
• paper could be improved in several ways . Several questionnaire . . 



respondents suggested tha't the process j>f 'formulating questions for ^ 
the director would be facilitated if some guidelines were provided 
by the instructor. These guidelines would obviously not focus on 
the ccmteht of the , questions , but would instead deal\*dth' such 
issues as format and scope. • •* 

Confusiort on the part of students during the simulation would 
also, be reduced if ' clearer guidelines were provided €oncerning the 

expected format for the other written: assignments. Distributing 

P • L V ' ' ' - 

sample proposals and finafVeports to. the class, would- probably be » 

helpful here. / , 

* » 
Rather than giving- all groups the same pre- and post-data to 

analyze, different sets of data appropriate to the. designs proposed 

by each group might be used. This procedure would link in a more, 

> ... . ^ j 

meaningful way- the task of preparing the final report \o that of 

' * * 

generating the proposals. Questionnaire responses indicated, that* 
•the major reason for the data-anajysis phase being so frequently 
cited as the least, valuable aspect- of the simulation was that many 
of the. students were quite disappointed at not being allowed^tO' " , 
♦"follow through on' their own designs. Since thevtask of. individual- 

* * * * 

izing data- sets woulg^signif icahtly increase the' instructor's .work- 
load, however, .it might only' be feasible* in courses where a* graduate' 
assistant, was^available. * - 

More could be learned from the* simulation if it incorporated* , 
in greater measure the bureaucratic processes, political conflicts, 

and budgetary.* constraints with which one must contend in real evalua- 

. < ' • >* . • • • 

tions. This. might be" achieved in part by' making t the initial setting 

de-script-Loiis more elaborate, th6ugh care must be taken to preserve 



the information gaps that students should fill-in themselves through' 
questions'. 0 tDne^ could also increase the simulations ability to capture 
the > ongoing dynamics of evaluatipn* by. introducing Jinjfoirma tion at' 
various points/throughout the simulation .which complicated <the> v 
'•scenario. Ih addition^- groups could be permitted \to question a 

' variety of individuals within and outsider of the agency as role- • . .V. 

*«• - * 

/flayed by the instructor, with the latter being free* to exhibit 

* v 7 ' ■ ' v 

varying degrees o^cooperativeness, s^fcrecy, trust, etc.. * < r * 
• " ■ If tf 

The richness of a given program, evaluation 'stimulation, j.n terms 

* # 

of the amount of realistic' detail characterizing it, will tend to • 
increase from 'semester to semester a.s the instructor adds a few new' 4 / 
touches' each time. . A particularly effective means of accomplishing 
-this is to generate .supporting 1 -^? terials for the ..simulatiqn in the* 

form of fictitious* annual*, reports,- staff handbooks, funding proposals, 

.*•*". ' 

osummary, minutes of staff' arid- board meetings*, etc.. Indeed, as this 
" ♦ - ' ; ' . > » - * 

• auxiliary material accumulate^" the simulation setting almost begins^ ■ 

"to take- 6n a life- of" $fi own: ? . .. /* / . . 

% ' » finally,' the systematic recording ~6f question's asked, by the f . 

• grou-ps wouldAissist* the "instructor, i~rT identifying, arfas of program 8 „ 

♦ . '* • ' ; . ' • 1 I - * • • ,. "' m 
evaluation that warrant' greater, attention in- course "lectures .--It". 

\ - • ' 

would be important her.e to , assess ihot. only, whether th£ appropriate 
domains of program evaluation we're being addressed by the questions , •' 
but also,-whether the questions were formulated- in .a .way which indicated 

en ad^qu^te understanding of j\6vt to solicit infonriation^within a 

'* - * f • \ ^ * 

giveji domain. Wheft 'the groups* questions a r e .exapin^d An % thds fashion, 

V the simulation becomes a method for increasing not only the student 1 

effectiveness in -the real world, bjut <the instructor^ ef fefc.tivenjss • 

* in'the classroom. as well.*. > * % : 
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Sdjriulatipn, Questionnaire Results: I 



Item 



\ . Me4n 
Resoonse 



1. 



3. 



4. 



A, 



7.- 



.8, 



How well forganized^did you feel 4 the simulated 
evaluation was?* flsvery well organized, 7=?very 
poorly Organizedj ' * - * * * 

How* interesting did you find the simulated 
evaluation? ' (l=very interesting, 7= not at all 
interesting] * t . +. ^ 

How. would you* rate the overall effectiveness of 
the simulated evaluation fcs a teaching device?, 
(l=very effective*, 7=not at all effective} 

If the simulated evaluation were re moved frdip the 
cburse (with no other procedure substituted), what 
'do ybu think would' be the effect on' the course? « 

* {l=would greatly • decreate^ its -quality , 7s would 
gVeatly increase its quality} " ■ w 

In the simulated evaluation an N fctt'terhpt was made to 
- approximate the difficulties, often *faced by a r<fa4 
evaluation team by not -initially revealing all of 
the information /aljout the hypothetical agency; that , 
: was needed.- to ^complete the evaluation* How useful 
* 'a -teaching device do you feel this procedure was'? 
pUvery-* useful , 7=not at .all, usef uii 

.Five questions concerning unknown facts about the' 
agency were allowed \before each of the proposals 
and final reports were due. How useful a teaching 

-.device dp you feel thi^ procedure was? - QUverV use- 
ful, 7=not at all usefufl I 

Three papers were required during the semester: an 

* initial proposal, a second proposal, and a final* 
evaluation report. How useful, a teaching device do 
^you feel this procedure was? .(livery useful, 7=not 
at all usfef uQ " ^ % * \ 

* Compared to the students who participated in ;a real 
evaluation during the semester., how- much do you think 
you* learned firom ' the simulated evaluation? fl=much ' 
more .than they &id r 7=much less than they dicO 



9. 



3.44 



.3.00 



2*19 



2; 13 



.2 J 



3*31 



2.25 



.4.25 



fpERJC . ' 



.Compared* tp a real evaluation, to what, extent dp you* 
feel. that the simulated evaluation technique, allowed . 
.fot fair, equitai>le grading^ of course wpxk? - (lfc simulated^ 
evaluation much more fai% ^simulated evaluation m,uch 
les.s; yaifl . ' ■ * ' 2*6l 



Appendix B , 



Simulation Questionnaire Results: .II , 



No ♦ of responses 



Item 



very * 
well 



moderately 
well < 



somewhat minimally 



not, 
at all 



To what extent do you- 
feel the simulated evalua- 
tion achieved each of the 
following objectives?- 

\. x To link the cl^ss 
lectures with an 
opportunity to apgly' 
these approaches and 
techniques in an 
evaluation setting. - 6 

2. To appreciate "first- 
hand" some of the dif- 
ficulties involved in * 
actually doift^an 

, e va lua t ion • / ^ 

3. .To* experience t the , 
process involved when r 
one works 'as a member 
of a group assigned 

9 the* task,. of conducting 
an evaluation. 6 



' 3 
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